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Abstract 

This paper considers a Markov-modulated duplication-deletion random graph where at each time 
instant, one node can either join or leave the network; the probabilities of joining or leaving evolve 
according to the realization of a finite state Markov chain. The paper comprises of 2 results. First, 
motivated by social network applications, we analyze the asymptotic behavior of the degree distribution 
of the Markov-modulated random graph. Using the asymptotic degree distribution, an expression is 
obtained for the delay in searching such graphs. Second, a stochastic approximation algorithm is presented 
to track empirical degree distribution as it evolves over time. The tracking performance of the algorithm 
is analyzed in terms of mean square error and a functional central limit theorem is presented for the 
asymptotic tracking error. 

Index Terms 

Complex networks, empirical degree distribution, giant component, Markov-modulated random graphs, 
power law, searchability, stochastic approximation. 

I. INTRODUCTION 

Dynamic random graphs have been widely used to model social networks, biological networks [1] 
and Internet graphs [2]. Motivated by analyzing social networks, this paper considers Markov-modulated 
dynamic random graphs of the duplication-deletion type which we now describe: 

Let n = 0, 1,2, . . . denote discrete time. Let 9 denote a discrete time Markov chain with state space 
{1,2,..., M}, evolving according to the M x M transition probability matrix A p and initial probability 
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distribution ttq. A Markov-modulated duplication-deletion random graph is parameterized by the 7-tuple 
(M, A p , ttq, r,p, q, Go). Here p and q are M-dimensional vectors with elements p(i) and q(i) £ [0,1], 
i = 1, . . . , M. p(i) denote the connection probabilities and q(i) denote the deletion probabilities. Also, 
r G [0, 1] denotes the probability of duplication step and Go denotes the initial graph at time 0. Go can 
be any finite simple connected graph. For simplicity we assume that Go is a simple connected graph 
with size Nq. The duplication-deletion random graph is constructed as follows: 

Algorithm 1 Markov-modulated Duplication-deletion Graph parameterized by (M, A p , ttq, r, p, q, Go) 
At time n, given the graph G n and Markov chain state 6 n , simulate the following events: 

Step 1: Duplication step: With probability r implement the following steps: 

• Choose node u from graph G n randomly with uniform distribution. 

• Vertex-duplication: Generate a new node v. 

• Edge-duplication: 

- Connect node u to node v. (A new edge between u and v is added to the graph.) 

- Connect each neighbor of node u with probability p(8 n ) to node v. These connection events 
are statistically independent. 

Step 2: Deletion Step: With probability q(6 n ) implement the following step: 

• Edge-deletion: Choose node w randomly from G n with uniform distribution. Delete node w and all 
edges connected to node w in graph G n . 

• Duplication Step: Implement Step 1. 
Step 3: Denote the resulting graph as G n+ \. 
Generate Markov state 9 n+ i using transition matrix A p . 

Step 4: Network Manager's Diagnostics: The network manager computes the estimates of the expected 
degree distribution. Denote the resulting graph as G n+ \. 
Set n — > n + 1 and go to Step 1 . 



For convenience in our analysis, assume that a node generated in the duplication step cannot be 
eliminated in the deletion step immediately after its generation. Also to prevent the isolated nodes, assume 
that the neighbor of a node with degree one cannot be eliminated in the deletion step. The duplication 
step (Step 2) is purely for convenience - it ensures that the graph size does not decrease. The Markov- 
modulated random graph generated by Algorithm [TJmimics social networks where the interaction between 
nodes evolves over time due to underlying dynamics such as seasonal variations (e.g., the high school 



friendship social network evolving over time with different winter/summer dynamics). In such cases, the 
connection/deletion probabilities p, q evolve with time. Algorithm Q] models these time variations as a 
finite state Markov chain 6 n with transition matrix A p . 

Context: Why is the degree distribution important? 

The expected degree distribution yields useful information about the connectivity of the random graph. 
For example, if a majority of nodes in the random graph have relatively high degrees, the graph is highly 
connected and a message can be transferred between two arbitrary nodes with shorter paths. However, if 
a majority of nodes have smaller degrees then for transmitting a message throughout the network, longer 
paths are needed, see [3]. Also, the degree distribution can be used to determine the existence of "giant 
component'd The existence of a giant component has important implications in social networks in terms 
of modeling information propagation in a social network and in human disease modeling [4], [5], [6]. If 
the average degree of a random graph is strictly greater than one then with probability one there exists a 
unique giant component [2] and the size of this component can be computed from the expected degree 
sequence. The average degree and the size of giant component is computed at each time as a measure 
of connectivity by the monitoring node. Another application of tracking the expected degree distribution 
is to estimate adaptively the "searchability" of the network. The searchability of a social network [7] is 
the average number of nodes that need to be accessed to reach another node. In this paper, we track the 
searchability of the network by means of tracking the expected degree distribution at each time. 

Main Results and Paper Organization: 

Notation: At each time n, let N n denote the number of nodes of graph G n . Also, let f n (i) denote the 
number of vertices of graph G n with degree i. Clearly J2i>i fn(i) = N n . Define the "empirical vertex 
degree distribution" as 

9n(i) = ^r-, forl<i<N n . (1) 

Note that g n (i) can be viewed as a probability mass function since g n (i) > and Y^iOnii) = 1- 
Let g n = E{g n } denoted the "expected vertex degree distribution" where g n is the empirical degree 
distribution defined in CO. 



'A giant component is a connected component with size 0{n) where n is the total number of vertices in the graph. 



Given the above Markov-modulated random graph, this paper presents three main results. 
Result 1: Asymptotic Degree Distribution Analysis of fixed size Markov-modulated duplication-deletion 
random graph 

Consider the sequence of finite duplication-deletion random graphs {G n }, generated by Algorithm [T] 
with r = 0. Clearly the number of vertices in the graph generated by Algorithm Q] with r = satisfies 
N n = Nq for n = 1, 2, . . . (The size of random graph is fixed.)- Assume that the Markov chain 6 n evolves 
according to a slow transition matrix A p = I + pQ, where Q is a generator matrix and p is a small 
positive constant. A novel degree distribution analysis is provided for the fixed size Markov-modulated 
duplication-deletion random graph in SecJII] Theorem 12.11 shows that for each 6 n = 9, the expected 
degree distribution of the finite random, g(8), can be computed from (fT2l . 

The asymptotic degree distribution analysis allows us to investigate the searchability and connectivity 
of the random graph generated according to Algorithm [T]as described in SecJIll Also, using the asymptotic 
degree distribution, the existence and size of the giant component in the random graph can be explored. 

Result 2: Tracking the Empirical Degree Distribution 
In Sec|llll we address the following two questions: 

• How can a network manager estimate (track) the empirical degree distribution using a stochastic 
approximation algorithm without knowledge of Markov ian dynamics? 

• How good is the estimate g n generated by the stochastic approximation algorithm (TJ]) when the 
random graph evolves according to Algorithm [7]? 

In Sec]]]]] we propose a stochastic approximation algorithm to estimate the degree of each node in random 
graph which can be modeled by Algorithm Q] Consider the finite Markov-modulated duplication-deletion 
random graph generated by Algorithm Q] with 7-tuple (M, A p , iro,p, q, r, Go) where r = 0. Suppose at 
each time n, noisy measurements, y n the empirical distribution of g n are obtained by the administrator 
of the social network. The network manager does not have information about the Markovian dynamics 
and deploys a non-parametric stochastic approximation algorithm to estimate the expected vertex degree 
distribution. More precisely, given these measurements y n , n = 1, 2, . . ., the network administrator aims 
to estimate the time varying expected vertex distribution g(6 n ). It deploys the following constant step 
size stochastic approximation algorithm: 

9n+l =dn + £ [Vn+l ~ ?n] (2) 

Here e > denotes a small positive step size. Eq. © is merely an exponentially discounted empirical 



distribution of the noisy node degree. Let g n = g n — E{g(#„)} denote the tracking error of the estimate 
of the empirical distribution of node degree. We present three results regarding the tracking performance 
of the degree distribution of the random graph: 

• 2-a. Mean square error analysis: Theorem 13. II in Sec jIII-Al shows that the mean squared of tracking 
error (the distance between E{g(#„)} and the estimated probability mass function (PMF) g n ) is of 
order of O + p + (Recall e is the step size of the stochastic approximation algorithm and 
p parameterizes the speed of the underlying un-observed Markovian dynamics). Derivation of this 
result uses error bounds on two-time scale Markov chains and perturbed Liapunov function methods. 

• 2-b. Weak convergence analysis: Theorem l3.2l in Sec illl-Bl shows that the asymptotic behavior of the 
stochastic approximation algorithm d2) converges weakly to the solution of a switched Markovian 
ordinary differential equation 

^- = -g(t) + g(e(t)l 9(0) =90- (3) 

• 2-c. Functional central limit theorem for scaled tracking error: How can the tracking error in the 
empirical distribution estimate be quantified? Sec JIII-Cl investigates the asymptotic behavior of the 
scaled tracking error. Similar to [8], it is shown that the interpolated scaled tracking error (between 
the expected and the estimated PMF) converges weakly to the solution of a switching diffusion. 
Let V}. = 9 fc -~ E ^|( gfc )} denote the scaled tracking error. Theorem 13.31 in Sec JUI-Cl proves that under 
reasonable conditions, the interpolated sequence of iterates, v e (t) = for k G [ke, {k + l)e) 
converges weakly to the solution of the following Markovian switched diffusion process 

du{t) = -v(t)dt + (si(0(t))) du, (4) 

where cj(-) is an W N ° -dimensional standard Brownian motion and E(0) S W N( > xNo is the covariance 
matrix. Eq. (01) (and Theorem I3.31 l are functional central limit theorems. The dynamics of the error 
in (0]) follow a Markov-modulated diffusion process. The covariance £(#(t)) for large t is used as 
a measure for the asymptotic convergence rate of the tracking algorithm. 
Note that the Markovian assumption only appear in our analysis, the stochastic approzimation algorithm 
© does not assume knowledge of the underlying Markov chain. (In [9], [10] this analysis falls under 
the class of analysis of a stochastic approximation algorithm with a Markovian hyperparameter.) 

Result 3: Power law component for infinite duplication-deletion random graph without Markovian 
dynamics 

SecJIV] extends the results of SecJII] and investigates the dynamics of the graph generated according 
to Algorithm Q] with r = 1 and when there are no Markovian dynamics, that is, M = 1. Since r = 1 



for n > 0, G n+ i has one more vertex compared to G n . In particular, since Go is an empty set, G n has 
n nodes, that is, N n = n. Theorem 14. 1 1 proves that the expected node degree distribution g n satisfies a 
power law as n — > oo. That is, 

log g n (i) = a — filogi as n — > oo 
where a and /3 are non-negative real numbers. The power law component, /5, satisfies 

(l+q)(p l3 ~ 1 +p/3-p) = l+/3q. (5) 

where p and q are the probabilities defined in Algorithm [T] The above result slightly extends [1], [11] 
where only a duplication model is considered. Theorem 14.11 parametrizes the degree distribution of the 
infinite duplication-deletion random graph without Markovian dynamics generated by Algorithm Q] by 
the power law component. Theorem 14.11 allows us to explore the searchability of the network and also 
the existence and size of the giant component of the infinite duplication-deletion random graph without 
Markovian dynamics. 

Related Works: 

We refer to [12], [13] for a comprehensive development of stochastic approximation algorithms. Here, 
the related literature on dynamic social networks is reviewed briefly. The evolution of random graphs is 
investigated in several papers, [14], [15]. The book [16] provides a detailed expositions of random graphs. 
The model of Pastor-Satorras et al.[ll] makes the basis for the model which is studied and generalized 
in this paper. In this model, at each time step, a new node joins the network. In the literature, it has 
been shown that the degree distribution of such network satisfies power law[17], [18]. In random graphs 
which satisfy the power law, the number of nodes with an specific degree depends on a parameter called 
power law component. A general complex graph generated by any arbitrary pure duplication, may not 
satisfy the power law. The power law distribution is satisfied in many other networks such as WWW- 
graphs, peer-to-peer networks, phone call graphs and various massive social networks (e.g. Yahoo, MSN, 
Facebook)[19], [20], [21], [22], [23], [24], [25]. The power law component describes asymptotic behavior 
of an online social network e.g. maximum degree, existence of giant component, diameter of the graph, 
and etc. [26] provides condition on the evolution of the graph to satisfy power law and shows that as a 
result of having an edge between nodes u and v, the resulting graph satisfies power law. 



II. Asymptotic Degree Distribution Analysis of the Fixed Size Markov-modulated 

Random Graph 

This section presents degree distribution analysis of the fixed size Markov-modulated duplication- 
deletion random graph. Consider the fixed size Markov-modulated duplication-deletion random graph 
generated according to Algorithm [TJwith 7-tuple (M, A p , 7ro,p, q, r, Go) where r = 0. The number of 
vertices in the graph generated by Algorithm [T] with r = is always Nq and the size of the graphs is 
fixed. Recall from Secfl] the state space of {9 n } is denoted as 

M = {1,2,...,M}, (6) 
and the transition probability matrix of 9 n is 

A p = I + pQ. (7) 

Here p is a small positive real number and so 9 n is a "slow" Markov chain. I is an M x M identity 
matrix, and Q is an irreducible generator of a continues-time Markov chain. Let q^ denote the elements 
of the generator matrix Q such that 

• (A) qij > if i ^ j and Vi, Ylj=i Qij = 0- F° r simplicity, we assume that the initial distribution 
7ro is independent of p. Q is irreducible^. 

Theorem 12.11 below proves that the expected degree distribution of the fixed size markov-modulated 
duplication-deletion random graph satisfies a recursive equation from which the expected degree distri- 
bution can be found. 

Theorem 2.1: Consider the fixed size Markov-modulated duplication-deletion random graph generated 
according to Algorithm [T] with 7-tuple (M, A p , 7i"o,p, q, r, Go) where A p = I + pQ and r = 0. Let 
lf n = E{<? n |0 n = 8}. The expected degree distribution of nodes in the fixed size Markov-modulated 
duplication-deletion random graph, ~g e n , satisfies the following recursion 

fn+l = (I + ^- o L'(0))f n , (9) 

2 The assumption of irreducibility implies that there exists a unique stationary distribution for this Markov chain, tv £ R Mxl 
such that 

tt' = n'A". (8) 



Iji 



where ' denotes transpose of a matrix and L(9 n ), with elements defined in (fTOl) . is a generator matrix 
(that is, each row adds to zero and each diagonal element of L(9 n ) is negative): 

j < i - 1 

q(e n )p(0nY- 1 + q(e n )(l +p(9 n )(i - 1)) j = i - 1 

iq(e n )p(6 n ) i - 1 (i- P {e n ))-q(e n )(i + 2+ P (6 n )i) j = i fon<i,j<N (io) 

# b )E)pW w (1 - P(^)) 2 + <?(W + 1) J = i + 1 

k q(e n ){ i L 1 )p{e n ) i - 1 (i-p(e n )y- i+1 j>% + i 

■ 

The proof is presented in Appendix |A] 

Theorem 12.11 shows that the evolution of the expected degree distribution in a fixed size Markov- 
modulated duplication-deletion random graph satisfies ©. Eq. (© can be re- written as 

f n+l =B'(6)f n , (11) 

where B(9 n ) = I + -±L(9 n ). Since L(9 n ) is a generator, for sufficiently large No, B(9 n ) can be 
considered as the transition matrix of a Markov chain. Hence, for each state of the Markov chain 9 n = 
9 G {1, 2, ... , M}, there exists a unique stationary distribution g(9) such that 

g(8) = B\8)g(9). (12) 

Therefore from (fT2l . the expected degree distribution of the fixed size Markov-modulated duplication- 
deletion random graph can be computed for each state of the underlying Markov chain 9 n = 9. Note 
that the underlying markov chain 9 n depends on the small parameter p. The main idea is that although 
9 n is time-varying but it is piecewise constant and since p is small parameter, it changes slowly over 
time. Also from d9j, the evolution of lf n depends on Our assumption throughout this paper is that 
p <S jf. This means that the evolution of lf n is faster than the evolution of 9 n or equivalently it can be 
said that lf n reaches its stationary distribution (g(9)) before the state of 9 n changes. 

Example: Searchability of a Network 

So far in this section, an asymptotic analysis of the degree distribution was presented for a random 
graph generated according to Algorithm Q] We now comment briefly on how the degree distribution can 
be used to investigate the searchability of the network. This also motivates the stochastic approximation 
algorithm presented in SecJIII] as will be described below. The search problem arises in a network when 
a specific node faces a problem (request) whose solution is at other node (e.g., delivering a letter to a 



specific person or finding a web page with specific information). Assume [7] that on receiving a search 
request, each node follows the following protocol: (a) It address the request if it or its neighbors have 
the solution; otherwise (b) it relays the request to one of its neighbors chosen uniformly. The objective 
is to find the expected search delay, that is, the expected number of steps until the request is addressed. 

Lemma 2.1: Consider the sequence of fixed size Markov-modulated duplication-deletion random graph 
obtained by Algorithm [Q {G n }, with (M, A p , iro,p, q, r, Go) where A p = 1+ pQ and r = and expected 
degree distribution g n . The expected search delay is 

A(iVo) = O (J^-) , (13) 
\d 2 -diJ 

as n — > oo where d\ = Ylf=i W anc * ^2 = Y^d=\ * 2 5n W- 

Proof: See Chapter 5 of [7] and recall that size of the considered random graph is Nq. ■ 
Lemma 12.11 implies that, if the empirical degree distribution of the possibly time-varying network can 
tracked accurately, then such an estimate can be used to track the searchability of the network. Also, 
using the estimated degree distribution and Lemma 12.11 we can address the following design problem 
as: How can p and q in Algorithm\l\be chosen so that the average delay does not exceed a threshold? 
Using the stochastic approximation algorithm in d2) (see SecJIII] below for the convergence proof), we 
can estimate the expected degree distribution, g n , and from that, we can compute d\ and di- Then, from 
Lemma 127X1 we can find the measure of searchability and compare it with the maximum acceptable average 
delay and modify the parameters of Algorithm Q] accordingly. We illustrate searchability in numerical 
examples given in SecjV] 

III. Estimating (Tracking) the Degree Distribution of the Fixed Size 
Markov-modulated Duplication-deletion Random Graph 

In SecJIII a degree distribution analysis is provided for the fixed size Markov-modulated duplication- 
deletion random graph generated by Algorithm [TJ with 7-tuple (M, A p , 7TQ,r,p, q, Go), where r = 0, Go 
is a simple connected grapeh of size Nq and A p is defined in In this section we assume that the 
empirical degree distribution of this graph, g n , is observed in noise by a network administrator. How 
can the network administrator track the expected degree distribution of the fixed size Markov-modulated 
duplication deletion random graph without knowing the dynamics of the graph? Suppose that the vertex 
distribution f n generated according to Algorithm [TJis measured in noise by the administrator of the social 
network. That is, the measurement is 

L = fn + U n . (14) 



Here, at each time n, the elements co n (i) of the noise vector are integer- valued zero mean random variables 
and Yli>i w nW — 0. The zero sum assumption ensures that /„ is a valid empirical distribution. In terms 
of the empirical vertex distribution, we can rewrite this measurement process as 

/ -\ fn(i) fn(i) r\ , 1 f\ 

that the vertex distribution g n of the graph G n generated according to Algorithm [T] is measured in noise 
by the administrator of the social network. That is, the measurement is 

Vn = 9n + e n (15) 

where e n = jj^. Recall that N n = Nq when r = 0. The normalized noisy observations from the 
monitoring node, y n , are used to estimate the empirical probability mass function of degree of each 
node. To estimate a time varying PMF, the following stochastic approximation algorithm with constant 
step size, e (where e denotes a small positive constant), is used to estimate the empirical probability mass 
function: 

9n+l = 9n + £ (Vn ~ 9n) ■ (16) 

Note that the stochastic approximation algorithm (fTol i does not assume any knowledge of the Markov- 
modulated dynamics of the graph. The Markov chain assumption for the random graph dynamics is only 
used in our convergence and tracking analysis. Our goal is to analyze how well the algorithm tracks the 
empirical node degree of the graph. This section studies the asymptotic behavior of the estimated degree 
distribution. Let E{g(9 n )} denote the expectation of g(0 n ) with respect to cr-algebra, Q, generated by 
{Uk, k < n}. First, we show that the difference between E{g(# n )} and g n , obtained by stochastic 
approximation, is bounded and the upper bound depends on e and p. 

A. Tracking Error of the Stochastic Approximation Algorithm 

Recall that the tracking error is g n = g n — E{5(6* n )}. Theorem 13. 1 1 below shows that the difference 
between sample path and the expected probability mass function is small - -implying that the stochastic 
approximation algorithm can successfully track the Markov-modulated node distribution given noisy 
measurements (We again emphasize that not knowledge of the Markov chain parameters are required in 
the algorithm). It also finds the order of this difference in terms of e and p. 

Theorem 3.1: Consider the random graph (M, A p , ttq,p, q, r, Go). Suppose that p 2 = o(eJ^]. Then for 



3 Note that in this paper, we assume that p = O(e), therefore p 2 = o(e) is a consequence. 



sufficiently large n the tracking error of the stochastic approximation (|2) is 

:2 



E l?n| 2 = O ( £ + p+ — I . (17) 



£ 



The proof of Theorem 13.11 is presented in Appendix |E] In the proof, the perturbed Liapunov function 
methods are used. As a corollary of Theorem 13.11 we obtain the following mean squares convergence 
result. 

Corollary 3.1: Under the conditions of Theorem 13.11 if p = O(e) we have 

E\g n \ 2 = 0{£). 



and therefore, 



limsupE|g n | 2 = 0. 



B. Limit System of Regime -Switching Ordinary Differential Equations 

Theorem 13.21 shows that the sequence of estimates generated by the stochastic approximation algorithm 
(fT6l ) converges weakly to the dynamics of a Markov-modulated ordinary differential equation. 

Theorem 3.2: Consider the Markov-modulated random graph generated by Algorithm Q] and the se- 
quence of estimates {g n } generated by stochastic approximation algorithm ([T6V Assume condition (A) 
holds, and p = 0(e). Define the continuous-time interpolated process 

sF{t) = 9n, £ (t) = 6 n t£ [ne, (n + l)e). (18) 

Then as e — > 0, (g 6 (■) , 9 £ (■)) converges weakly to (g{-),0(-)) such that 6 is continuous-time Markov 
chain with generator Q and g(-) satisfies the Markov-modulated ordinary differential equation (ODE) 

^ = -9(t)+mt))> 5(0) = 50, (19) 

where g(9) is defined in (fT2l . ■ 
Note that (fT9l ) is a Markov-modulated ordinary differential equation. The above theorem asserts that the 
empirical measure obtained by stochastic approximation algorithm (fT6l ) converges weakly to Markovian 
switched ODE ( fT9l ). As mentioned in Sec|]] this is unusual since typically in averaging of stochastic 
approximation algorithms, convergence occurs to a deterministic differentia equation. The intuition behind 
that the estimates obtained by (fT6l ) converges to a Markov-modulated ODE (rather than a deterministic 
ODE) is that the Markov chain (with transition matrix I + pQ ) evolves on the same time scale as the 
stochastic approximation algorithm with step size e (when p = 0(e)). If the Markov chain evolved on 



a faster time scale, then the limiting dynamics would indeed be a deterministic ODE weighed by the 
stationary distribution for the Markov chain. If the Markov chain evolved slower than the dynamics of 
the stochastic approximation algorithm, then the asymptotic behavior would also be a deterministic ODE 
with the Markov chain being a constant. 

C. Scaled Tracking Error 

The following theorem studies the behavior of the scaled tracking error between the estimates generated 
by the stochastic approximation algorithm ([ToT i and the expected degree distribution and proves that this 
error should also satisfy a switching diffusion equation. Theorem 13.31 gives a functional central limit 
theorem for this scaled tracking error. Let = iizI^DMfbsR denote the scaled tracking error. 

Theorem 3.3: Assume condition (A) holds. Define v £ {t) = v\- for t G [ke, (k+l)s). Then (v e (-), 9 £ {-)) 
converges weakly #(•)) such that u(-) is the solution of the following Markovian switched diffusion 
process 



where ui(-) is an R^ -dimensional standard Brownian motion. The covariance matrix, £(#), in d20l > can 
be explicitly computed as 



Here, D{9) = diag(#(0, 1), . . . ,g(6,N )) and Z{6) = {I - B{9) + l<f(0)) where B(9 n ) and g(0) are 



For general switching processes, we refer to [27]. In fact, more complex continuous-state dependent 
switching rather than Markovian switching was considered there. Eq. (|2TT > reveals that the covariance 
matrix of the tracking error depends on B{9) and g(9) and consequently on the parameters of p and q 
of the random graph. Recall from SecJII]that B{6) is the transition matrix of the Markov chain which 
models the evolution of the expected degree distribution in Markov modulated random graph and can be 
computed from Theorem 12.11 We can interpret the covariance matrix in terms of searchability of the graph 
defined in SecjE] SecJV] provides numerical examples that show that the trace of the covariance matrix 
E(0) is proportional to the searchability of the graph generated by Algorithm Q] Numerical examples in 
SecjV] also show that the trace of covariance of the tracking error is proportional to the average degree 
of nodes. 




(20) 



E(0) = Z{9)'D{9) + D{9)Z{9) - D{9) - g(0)tf(0). 



(21) 



defined in (JT]) and (fT2l . respectively. 



IV. Discussion and Extension: Power Law Component for Infinite 
Duplication-deletion Random Graph Without Markovian Dynamics 

In SecJIIl a degree distribution analysis is provided for the fixed size Markov-modulated random graph 
generated according to Algorithm [T] with r = 0. This section extends the results of SecJII]to the infinite 
duplication-deletion random graph without Markovian dynamics. Here, we investigate the random graph 
generated according to Algorithm [TJ with r = 1 and when there are no Markovian dynamics, that is, 
M = 1. Since r = 1 for n > 0, G n+ i has one more vertex compared to G n . In particular, since Go is an 
empty set, G n has n nodes, that is, N n = n. In this section, employing the same approach used in the 
proof of Theorem 12.11 it is shown that the infinite duplication-deletion random graph without Markovian 
dynamics generated by Algorithm [T] with r = 1 satisfies a power law and an expression is derived for 
the power law component. Let us first define the power law: 

Definition 4.1 (Power Law): Consider the infinite duplication-deletion random graph without Marko- 
vian dynamics generated according to Algorithm Q] with 7-tuple (M, A p , tto,P, q, r, Go). Let rij. denote 
the number of nodes of degree k in a random graph G n . Then G n satisfies a power law distribution if 
rik is proportional to kr^ for a fixed /3 > 1 : log = a — j3 log k, where a is a constant. j3 is called 
power law component. 

Theorem 4.1: Consider the infinite random graph with Markovian dynamics G n obtained by Algorithm 
Q] with 7-tuple (1, 1, 1, l,p, q, Go) with the expected degree distribution g n . As n — > oo, G n satisfies a 
power law. That is 

^og g n (i) = a- /3 log i, (22) 
where the power law component, (3, can be computed from following equation. 

(l + q )(pf^ +p p-p) = l + [3 q , (23) 

where p and q are the probabilities defined in duplication and deletion steps. ■ 
Remark 1. Outline of Proof: The proof of Theorem 14. 1 1 which is presented in Appendix iBl consists of 
two steps: (i) finding the power law component and (ii) showing that the degree distribution converges 
to a power law as n — > oo. To find the power law component, we derive a recursive equation for the 
number of nodes with degree i + 1 at time n + 1, f n +i(i + 1)> in terms of degree of nodes in graph G n . 
Then, this recursive equation is rearranged to equation for the power law component. To prove that the 
degree distribution satisfies a power law, we define a new parameter h n (i) = - £)fc=i E{/ n (fc)} and we 
show that linin^oo h n (i) = J2k=i Ck~^ where /3 is the power law component computed by the solving 



the recursive equation. Theorem 14.11 asserts that the infinite duplication-deletion random graph without 
Markovian dynamics generated by Algorithm Q] satisfies a power law and provides an expression for the 
power law component. The significance of this theorem is that it ensures that with use of one single 
parameter (the power law component), we can describe the degree distribution of large numbers of nodes 
in graphs that model social networks. 




Fig. 1. The power law component for the non-Markovian random graph generated according to Algorithm Q~] obtained by (1231 
for different values of p and q in Algorithm [T] 

Remark 2. Power Law Component: Let /3* denote the solution of (|23T ). Then the power law component 
is defined as (3 = max{l,/3*}. FigQ] shows the the power law component and /3* versus p for different 
values of probability of deletion, q. As can be seen in FiglH the power law component is increasing in 
q and decreasing in p. 

V. Numerical Examples 

In this section, numerical examples are given to illustrate the results from SecJII] SecJTITl and SecJTVl 
The main conclusions are: 
(i) The infinite duplication-deletion random graph without Markovian dynamics generated by Algo- 
rithm Q] satisfies a power law as stated in Theorem 14.11 This is illustrated in Example 1 below. 



(ii) The degree distribution of the fixed size duplication-deletion random graph generated by Algorithm [T] 
can be computed from Theorem 12.11 When No (the size of the random graph) is sufficiently large, 
numerical results show that the degree distribution satisfies a power law as well. This is shown in 
Example 2 below. 

(iii) The estimates obtained by stochastic approximation algorithm ([Tol follow the expected probability 
distribution precisely without information about the Markovian dynamics. This is illustrated in 
Example 3 below. 

(iv) The larger the trace of the asymptotic covariance of the scaled tracking error, the greater the average 
degree of nodes and the searchability of the graph. This is illustrated in Example 4 below. 

Example 1 : Consider an infinite duplication-deletion random graph without Markovian dynamics gen- 
erated by Algorithm Q] with p = 0.5 and q = 0.1. Theorem 14. 1 1 implies that the degree sequence of the 
resulting graph satisfies a power law with exponent computed using (l40b . Figj2] shows the number of 
nodes with specific degree on a logarithmic scale for both horizontal and vertical axes. It can be inferred 
from the linearity in Figfl] (excluding the nodes with very small degree), that the resulting graph from 
duplication-deletion process satisfies a power law. As can be seen in FigJU the power law is a better 
approximation for the middle points compared to both ends. 




Fig. 2. Illustration of Theorem 14.11 The degree distribution of the duplication-deletion random graph satisfies a power law. 
The parameters are specified in Example 1 of SeclVl 



Example 2: Consider the fixed size duplication-deletion random graph obtained by Algorithm Q] with 
r = 0, iVo = 10, p = 0.4, and q = 0.1. (We consider no Markovian dynamics here to illustrate 
Theorem 12.11 ) Fig. |4] depicts the degree distribution of the fixed size duplication-deletion random graph 
obtained by Theorem 12.11 As can be seen in Fig. |4] the computed degree distribution is close to that 
obtained by simulation. The numerical results show that the degree distribution of the fixed size random 
graph also satisfies a power law for some values of p when the size of random graph is sufficiently 
large. Fig. [3] shows the number of nodes with specific degree for the fixed size random graph obtained 
by Algorithm Q] with r = 0, Nq = 1000, p = 0.4, and g = 0.1 on a logarithmic scale for both horizontal 
and vertical axes. 




Fig. 3. The degree distribution of the fixed size duplication- 
deletion random graph satisfies a power law when No is suf- 
ficiently large. The parameters are specified in Example 2 of 
Sec|V] 



Fig. 4. Illustration of Theorem 12.11 The degree distribution of 
the fixed size duplication-deletion random graph. The parameters 
are specified in Example 2 of SecfVl 



Example 3: Consider the fixed size Markov-modulated duplication-deletion random graph generated by 
Algorithm Q] with r = and No = 500. Assume that the underlying Markov chain has three states, M = 3. 
We choose the following values for probabilities of connection and deletion: state (1): p = q = 0.05, 
state (2): p = 0.2 and q = 0.1, and state (3): p = 0.4, q = 0.15. The sample path of the Markov chain 
jumps at times n = 3000 from state (1) to state (2) and n = 6000 from state (2) to state (3). As the state 
of the Markov chain changes, the expected degree distribution, g(9), obtained by (fT2l evolves over time. 
The corresponding values for the expected degree distribution (for i = 3) are shown in FigJ5] by a dotted 
line. The estimated probability mass function, g n , obtained by the stochastic approximation algorithm 



(O is plotted in Fig|5] using a solid line. The figure shows that the estimates obtained by the stochastic 
approximation algorithm ([ToT i follow the expected degree distribution obtained by ([P2l precisely without 
any information about the Markovian dynamics. 

Example 4: Consider the fixed size Markov-modulated duplication-deletion random graph obtained by 
Algorithm Q] with M = 91 and r = and N = 1000. For each value of p(9) = 0.04 + 6 x 0.01, 9 G 
{1,2,..., 91} and q G {0.05, 0.1, 0.15, 0.2}, we compute L(8) from (TTOl i and consequently the stationary 
distribution, g(6), from (fl~2l . As expected, the stationary distribution does not depend on q because 
only the deletion step in Algorithm Q] occurs with probability q. From g(9), we compute the average 
degree of nodes, d\. Figj6] shows the average degree of nodes versus the probability of the connection 
in Algorithm [T] As can be seen in Figj6] with increasing the probability of connection in Algorithm Q] 
the average degree of nodes in the graph (which is a measure for the connectivity of the graph, see [2]) 
increases. 




Iteration Probability of connection, p. in Algorithm 1 



Fig. 5. Illustration of Theorem 13.11 The estimated proba- 
bility mass function obtained by the stochastic approximation 
algorithm d!6t follows the expected probability distribution 
precisely without information about the Markovian dynamics. 
The parameters are specified in Example 4 of SecfVl 



Fig. 6. The average degree of nodes (as a measure of 
connectivity) of the fixed size Markov-modulated duplication- 
deletion random graph obtained by Algorithm Q] for different 
values of the probability of connection, p, in Algorithm Q] The 
parameters are specified in Example 4 of SecfVl 



Then for each value of p(0) = 0.04 + 9 x 0.01, 9 G {1, 2, . . . , 91} and q G {0.05, 0.1, 0.15, 0.2}, the 
covariance matrix is computed using ©. FigJT] depicts the trace of the covariance matrix, trace (£(#)), 
for each value of p and q versus the corresponding average degree of nodes (for each value of p). As 
can be seen in FigJTl the trace of the covariance matrix is larger when the average degree of nodes is 



higher (the graph is highly connected). 

Recall from Lemma 127X1 the order of delay in the searching problem can be computed by X(Nq) = 
O Knowing the degree distribution g(9), d\ and (i 2 can be computed for each value of p G 

\ 0,2 — 11! ) 

{0.05, 0.06, . . . , 0.95}. FigH]shows the trace of the covariance matrix versus ( - dl - ) as a measure of the 

\d 2 —di J 

searchability for each value of q G {0.05, 0.1, 0.15, 0.2}. As can be seen in FigJU the trace of covariance 
matrix is larger when the order of delay in the search problem in ( TT3T ) is smallec ■ 
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Fig. 7. The trace of the covariance matrix of the scaled tracking 
error, trace (E(#)), versus the average degree of nodes as a 
measure of connectivity of the network. The parameters are 
specified in Example 3 of SeclVl 



Fig. 8. The trace of the covariance matrix of the scaled 
tracking error, trace (E(#)), versus the order of delay in the 
searching problem as a measure of searchability of the network. 
The parameters are specified in Example 3 of SecfVl 



VI. Conclusion 

This paper analyzed the dynamics of a duplication-deletion graph where at each time instant, one node 
can either join or leave the graph (An extension to the duplication model of [1], [11]). The power law 
component for such graph was computed using the result of Theorem 14.11 Also a Markov-modulated 
random graph was proposed to model the social networks whose evolution changes over time. Using the 
stochastic approximation algorithms, the probability mass function of degree of each node is estimated. 
Then, an upper bound was derived for the distance between the estimated and the expected PMF. As 
a result of this bound, we showed that the scaled tracking error between the expected PMF and the 



4 This means that the target node can be found in the search problem with smaller number of steps. 



estimated one weakly converges to a diffusion process. From that, the covariance of this error can be 
computed. Finally, we presented a discussion on application of this work in controlling a social network 
using the degree distribution obtained by stochastic approximation. In this case it is assumed that the 
network manager observes the degree of active users and this observation is noisy due to the activity 
profile of users. Using the estimated degree distribution, the network manager can track the level of 
connectivity (by computing the orders of size of giant component) and the searchability of the network 
(by computing the order of delay). 

Appendix 

A. Proof of Theorem I2.il 

Proof: To find the degree distribution of nodes, we find a relation between the number of nodes 
with specific degree at time n and the degree distribution of the graph at time n — 1. Given the resulting 
graph at time n, we are trying to find the expected number of nodes with degree i + 1 at time n + 1. 
The following events can occur that result in a node with degree i + 1 at time n + 1: 

• A node with degree i is chosen at the duplication step as a parent node. In this case, there will be 
another edge connecting the new node to the parent node in the edge-duplication step. Probability 
of choosing a node with degree i is ^f^-. If a node with degree i is not chosen itself but one of 
its neighbors is selected as parent node, there is also a chance for this node to have another edge 
(with probability of p). This node has i neighbors therefore, the corresponding probability is p.i. 
So the probability that the degree of such node increases by 1 in the duplication step is Also, 
in the deletion step nor this node neither any its neighbors should be selected in the edge-deletion 
step. With the same discussion, the associated probability is ^1 — <? ^ 1 ^ . If deletion step occurs, 
another node is generated and connected to the graph as described in deletion-step in SecU Nor 
this node (the node with degree i) and none of its neighbors should be selected in this step. So the 
probability that this node remains unchanged after deletion step is: ^1 — \ 

• A node with degree i + 1 at time n does not change during duplication and deletion processes. To 
be unchanged in both duplication and deletion steps, this node or any of its neighbors should not 
be chosen in both duplication and deletion steps. The probability of being unchanged during these 
processes for an specific node can be computed from ^1 — g ^ +2 ^ +9 C L+p ^ +1 ^) \ (l — p ^^ +1 ^ and 
total number of such nodes at time n is f n (i + 1). 

• The degree of the most recently generated node (in the vertex- duplication) increases to i + 1 in the 
edge-duplication step. This means that, this node is connected to "i" neighbors of the parent node 



and remains unchanged in the deletion step. The probability of this scenario is 

(i - "• +2,+ t +i " ,+1)) ) ^i/.WWi-j^. 

• A node with degree i + 2 remains unchanged in the duplication step and one of its neighbors is 
eliminated in the deletion step. The probability of this event is q fl ~~ P ^a^ +1 )- 

• The degree of the node generated in the deletion-step increases to i + 1 (As described in SecJH in 
deletion-step to maintain the total number of nodes, a new node is generated and connected to the 
graph). The probability of this scenario is qJ2j>i jrfn{j) —vV" 1 - 

• A node with degree i remains unchanged in the duplication step and the same node or one of its 

neighbors selected in the duplication part of the deletion step. The corresponding probability is 

q(i+P'Q / _ i+ P i \ 

N n y 1 N n ) 

• The degree of a node with i + 1 neighbors increases in the duplication step and one of its neighbors 
is eliminated in the deletion step. The corresponding probability is q (jy^j ^^((i+iHiLl^ 

Let Q denote the set of all arbitrary graphs and F n denote the sigma algebra generated by graphs 
G t ,t < n. Considering the above events that result in a node with degree i + 1 at time n + 1, the 
following recurrence formula can be derived for the conditional expectation of f n +i(i + 1): 

e(/„ +i(j + w = (i - , e±MM£fi±l») (, - £2+£±i) M + 1) 

1 -« fi±n ^)( 1 £r)/.M " 



[i + 2\ f p(i + 2) + 1\ , , 



A r n / V N n 

q(l+pi) ( 1 +pi\ , ... 



i + 2\ + + 



/«(» + !)■ (24) 



Let f n (i) = E{/ n (i)|6> n = 9}. By taking expectation of both sides of (l24l ) with respect to trivial sigma 
algebra {$7,0}, the smoothing property of conditional expectations yields. 



W« + 9 ^ ) \1 K ) /„(< + 

+ { 1 - q N n ) {—) fM 



1) 



+ 1 



( ' + * + ( £* + 1) s£*M(fl*'-'> 



yj-» 



Assuming that size of the graph is sufficiently large, each term like ' can be neglected for large N n . 
So (l25l) can be re-written as 



/n+l(* + !) = I 1 ^ I /n(* + !) 

+ [ K ) /„(•) + ffW ) /„(« + 2) 

+ £ QP(^1) ! (1 " P(^n + l)) J - i . (26) 

Using (|25T ). we can write the following recursion for the (z + l)-th element of ~g 9 (n + 1). 

9n+l(* + 1) = I -fi— I Sn(* + 1) 

+ I Jv n+1 J + 9W fej ^ + 2) 

+ *W £ JT~fnU) ft W)*(l - (27) 



Since the probability of duplication step r = 0, the number of vertices does not increase. Thus, 
N n = Nq and (1271 ) can be written as 

f n+1 (i + 1) =(l - i- (q(9)(i + 2) + q(9)(p(9)(i + 1) + 1)) )f n (i + 1) 

+ w ( (1 + pyw^ti® + Wo qm + 2) ^ n{i + 2) 

+ ]jrq(0)J2?nU) ( J )p(e)\l-p(9)y~\ (28) 

From d28j, it is clear that the vector g e (9 n+ i) depends on elements of g e (9). In a matrix notation, 
(l28l) can be re-arranged as 

^ +1 = (/+^M, (29) 

where L(9 n ) is defined as (flOl ). 

To prove that L(9 n ) is a generator, we need to show that la < and Yli^i ^ki = 0. 

No 

Y Zfci = - (q(0 n )(k + 1) + ?(0„)(1 + p(0„)fc)) + (1 + p(9 n )k)q(9 n ) 
i=i 



+q {9 n )k+ q {9 n ) y, { i k _\{9 n r 1 {i-p{9n)) k - 

k<i-l ^ ' 

= - q {9 n ) + q{9 n ) Y ( i k _\{9 n ) l '\l- P {9 n )) k - l+1 . (30) 



k<i-l 

Let m = i — 1. (l30b can be rewritten as 

N k 



/ l. \ 

X> = - + #„) J] MOT -p(^)) fc - m 

i=l m=0 ^ ' 

») + *i(i-*))'E(I)(i^) m (3D 



m=0 



We know that YT m =o y« m = (l + BMO <EQ> can be written as 

N 



X> = -q(0n) + q(9 n )(l-p(9n)) k ( x.p^) ) 

= 0. (32) 



Also it can be shown that if g(0„) < ^feff^ , then < 0. 



B. Proof of Theorem \4.1\ 

Proof: To prove Theorem 14.11 we first compute the power law component, j3, and then we prove 
that the expected degree distribution converges to the power law distribution with component f3. Let 
f n (i) = E{/ n (i)}. Similar to (l24l . f n (9 n ,i) can be written as 

W« + 1) = [1 ~ 1 Wn ) [I ^ ) /»(< + 1) 

/ (i + l) + (l+pi) \ fl+ P i\ 1 , 

+ { 1 - q N n J {— fM 



+ 1 



(i+2) + ( ^ y(i+1)) )g ^ o)( ^ (i _ p) 

+ «E^7»0')(^-(i-rt 



g(l +_gjO ( _ l+pi \ -j 



(0 



+ 9 Mf^ 7 , (j + 1) . (33) 



N n J \ N n 

To compute the power law component, we can heuristically assume that f n (i) = ait as N n = n goes 



to infinity (we will prove this precisely later on this section). Therefore, each term like can be 



neglected as n approaches infinity. So ( f33T > can be re-written as 

W» + 1) = \1 Wn j /„(! + 1) + ^ J /nW 

+ Q N£\ J n (i + 2) + (1 + «) £ ^7n(i) ('W - P)^- (34) 
Substituting / T (j) = Oj-t and iV n = n in (l34l) yields 

a i+ i(n + 1) =a i+ i?i - a i+1 (jl + p(i + 1))(1 + g) + g(? + 2)) + (1 + g)(l + pi)oj + g(i + 2)a i+2 



+ (l + g)£a i m P i (l-p)'-\ (35) 
Taking all terms with <2j + i to the left hand side, we have 

Oi+l (l + (1 + 9) (1 + P(i + 1)) + ?(« + 2))) =(1 + q) ^(1 + j»)oj + £ aj Qp l (1 - p)'"* 

+ g(* + 2)a i+2 . (36) 



Dividing both sides of (1361 ) by a-i yields 

(i + (i + q) (i + p( i + 1)) + g(i + 2 ))) =(i + q ) i (i + P i) + y; a -t (Afti - pf 

\ 3>l 

+ q(i + 2)^. (37) 

dj 

Solving Equation d36l ) for a i; we can complete the proof of Theorem 14. 1 1 The following lemma whose 
proof can be found in [1] is used to solve the recurrence relation for cij. 
Lemma 1.1: 

j>i x ' x ' 

Proof: The proof is presented in Appendix [C] ■ 
To solve (l36l ) for ctj, we can further assume that aj = Ci"' 3 [2]. Therefore, = (^j 2 ) ^ 

f 1 - j) + + + + 9(* + 2)) =(l + g)(l+^ + p /3_1 ) 

+ (i) + ,( ! + 2)(l-f). (39, 

Neglecting the O (A) terms, yields 



(l + (/)(/- 1 +p/3-p) = l + /3 ( ?. (40) 

Note that the proof presented above depends on few assumptions. To give a rigorous proof, the 
succeeding steps should be followed as described in [2]: 

• First, we need to show that the limit lim n _ > . 00 -~E{f n (i)} exists. 

• Let aj be the solution of d36l ) such that Yli^i a * = 1 an( ^ a o = 0> then it is needed to show that 

lim -E {/„(»)} = Oi. (41) 

n— >oo 7j 

• Finally, we should show that aj is proportional to where /3 is the root of (l40l . 

To complete the proof we define new function as follows h n (i) = A Y^k=i E{/n(^)} which can be 
described as CDF of degree of each node in random graph. It is sufficient to show that for all i > 0, 

i 

lim h n (i) = y^a k (42) 
k=X 

where cij is the solution of (l36l) . It is obvious if (l42l holds, h n (i) — h n (i — 1) = Oj and thus 

lim -E{/ n (i)} = Oj 

n— >oo 71 



(as presented in (|4~T1)). The following lemma gives a recurrence formula to compute the value of h(n+l, i). 
Lemma 1.2: 

h n+1 (i) = D n+1 (i)h n (i) + B n+1 (i)h n (i-l) + C n+l (i)h n (i + l) + ^^ £ h n (j)F(j,i-l,p), (43) 

71 3>i-l 



where 

n 



Ah-i(0 

B n+ \{i) 
C„+i(i) 



(<z(i + 2) + (l + g)(p» + l 
n + 1 

(l + q)(l + pi) 
n + 1 

g(j + 1) 
n + 1 ' 



fc=0 ^ ' fc=0 ^ ' 



This lemma can be proved by induction. The complete proof can be found in Appendix [D] The recursive 
equation presented in Lemma 11.21 is used later to prove that the degree distribution converges to a power 
law. 

Lemma 1.3: Let Sj = Y?k=i a * an< ^ 

w (n) = sup^, (44) 

where h n (i) satisfies (|43l . Then the limit lim^oo w(n) exists and we have lim-n^^ ui(n) = 1. 

Sketch of the proof: Knowing that /i n (i) satisfies the recurrence formula (143V the proof is similar 
to [2]. Plugging i = n in (l44l yields w(n) > > j- > 1. Using the Lemma [L2l and similar to [2], 

it can be shown that uj{n + 1) < oj(n). u(n) is bounded and decreasing, so the limit of lrnx^oo u(n) 
exists. To show limj^oo oj(n) = 1, we assume that lim n _> 00 uj(n) = c. It can be shown that if c ^ 1, 
w(n) < 1 is violated. Thus c = 1 and the proof is complete. 



C. Proo f of Lemma I7.il 
Proof: 

3>i Qi W j>i J W 

7^ 3 \J-y 



(i+<4>)»TC + r>-> 1 

v 7 fc=0 v 7 

+ tfpP-*- 1 = (l + (45) 



7). Proof of Lemma 17.21 

We prove the lemma by induction on i: 
For i = 1: It is sufficient to show that: 
h(n + 1,1) = 79 n+ i(l)/i(n, 1) + C„+i(l)/i(n,2) + ^ h{n,j)F(j, Q,p). Also using the definition 

of F(j, i,p), we can rewrite F(j, 0,p) as (1 — p) J — (1 — The number of nodes with degree one 

at time n + 1 can be written as following 

E{/(n + 1, 1)} = f 1 - + + E{/„(1)} + ^E {/n (2)} 

\ n J n 

+ (i+ g )y;-E{/ n (j)Ki-p) i - (46) 

Note that (l46l ) is slightly different from the general equation for each i, (l34l . Because as described 
in Sec|]J neighbors of a node with degree one cannot be eliminated from the graph to maintain the 
connectivity in the graph. Therefore, a node with degree one can change in the deletion step if that node 



is selected in the deletion step (with probability q). Using (l46l ). h(n + 1, 1) can be written as 

% + l,l) = -^-t E{/(n + l,l)} 
n + 1 

i ^ _ (I+iHI+i^ W (1)} + ^ E{/7i(2)} 



n+lyy n / n 

+ ^iEv e{Wj)}(1 - j,)j; (47) 

We know that h(n,0) = for all n. Using the definition of h(-, •) and (l46l ). (l47l) can be re-arranged 
as follows 

h(n + 1, 1) =-l^( n - ((1 + +p) + g))/i(n, 1) + ^(/i(n, 2) - h(n, 1)) 

+ (1 + q) J>(n,i) - /i(n,j - 1))(1 

j>l ' 

=^T ( (n-(39+(l + g)(l+p)))/»(n,l) + ^/»(n,2)^ 

+ ~t4 Z>( n >-?) " " 1))(1 (48) 

n + 1 

Sj>i(M n ' i) ~~ h(n,j — 1))(1 — p) J can be written in terms of the F(j, i,p). 

J>(n,j) ~ Hn,j - l))(l- P y = X>(«J)(1 -pf ~ J>(n,j - 1)(1 - P y 
i>i j">i j>i 

= J>(n,i)(l - J>(n, -p) j + 1 
i>i j>i 

= ^Th(n,j)((i-py-(i- P y +l ) 

i>i 

= ^%,i)F(j,0,p). (49) 

Substituting (09]) in (08]) yields 



fc(n + l,l) = ^-y (n- ((l + g)(l+ p) + 3g))/i(n,l) + ^%,2) + (1 + g) J] h(n,j)F{j,0,P) 



+C n+1 (l)/i(n,2) + i±I V/i(n,i)F(i,0,p). (50) 
n + 1 

i>i 



Thus (1431) holds for i = 1. Now it is assumed that (1431) holds for i = k, we want to show that it also 



holds for i = k + 1. 

E { /<„ + 1, * + ,)} _ f 1 - + + (l + «)(p(* + l) + lA Mf(n, k + 1)} 



+ 



(1 + g) l 1+ ^ )E {/ ^) }+ (#±j))E { / n ( fc + 2)} 

+ (i + ,)^M^( J \ fc (i-pr fc . (3D 

from definition of h(n, k), we have : E{/ n (/c)} = n (/i(n, fe) — h(n, k — 1)). Eq. d5TT > can be re- written 
as follows 



E{/(n + 1, A; + 1)} = (n - (q(k + 2) + (1 + q)(p(k + 1) + l) J J (h(n, k + 1) - h(n, k)) 

+ {l + q)(l+pk)(h(n,k) -h(n,k-l)) + q(k + 2) (h(n, k + 2) -/t(n,fc + l)) 
+ + (Mn,i) - - 1)) ( 3 X k {l- P y- k . (52) 



Using the Abel summation identity, and knowing that 

k / .\ k 



F(j,k, P ) = ( J k )p k (i-py~ k - E ( J t X V (1 -p) j+1 ~ k i 

i — n V / i — n \ / 



k=0 s ' fc=0 

the last term can be written as 



£ - M^i - 1)) ({V (i (53) 

= E((^ fc ( 1 -^- ^Vy^-Py^) ~P k h(n,k-l) 

= -p k h(n,k - 1) +£%, j){F(j,k,p) - F(j,k- l,p)). (54) 
i>fc 

Substituting (|53]> in ([52]> yields 



E{/(n + 1, k + 1)} =/i(n, k + 2)(q(k + 2)) + h(n, k + l)(n- (2q(k + 2) + (1 + q)(p(k + 1) + 1)) J 

+ h(n,k)((l + q)(2+p(2k + l)) 

+ q(k + 2) - + /i(n, k - 1)(1 + g)(-l -pk - p k ) 



+ (1 + q)J2h(n,j)(F(j,k,p) - F(j,k - l,p)) . (55) 

The value of h(n + 1, k + 1) can be computed using A; + 1) and E{/ n (/c + 1)} as follows 

hCn + 1, fc + 1) = h(n + 1, &) + — - E{/(n + 1, k + 1)}. (56) 

n + 1 



+ 



Eq.(l55l) gives an expression for E{/(n + 1, Ar + 1)} in terms of the value of h(-, •) at time n. Substituting 
(l55T ) in (1561 ) gives a recursive equation for computing h{n + 1, A; + 1): 

/i(n + 1, fe + 1) =/i(n + 1, k) + : -E{/(n + 1, fe + 1)} 

n + 1 

=D n+1 (k)h(n, k) + B n+1 (k)h(n, k - 1) + C n+ i/i(ra, fc + 1) 
n + 1 

i>fc-i 

— '— ( /i(n, fc + 2)( 9 (fc + 2)) + /i(n, k + 1) 
n+l\ 

- (2q(k + 2) + (l + q)(p(k + l) + l))^ 
+ h(n, k) ((1 + g)(2 +p(2k + 1)) + fc(n, k - 1)(1 + g)(-l - pA; - p fc ) 

+ (l + g)2 /l (",J')( F C?'. fc .P) -FO'^-l.P)) )• (57) 

j>fc / 

We assume that (l34l ) holds for i = so substituting the values for D n+ \{k), B n+ \{k), and C n +i{k) 
from (O in ([57]) yields 

/ fjfe + 2) \ In- (q(k + 3) + (1 + q)(p(k + + 

h(n + l,k + l) =h(n,k + 2) IT£2Lfl) +h{n,k + l) ^ 

\ n + 1 J \ n + 1 

+ h(„, k) ( (1 + 1 f + '» ) + l±i E „(„, J)(Fa M ) . (58, 
\ n + 1 / n + 1 

(l58l)can be written as follows 

fc(n + 1, k + 1) =Ah-i(A; + l)h(n, k + 1) + Sn+iO + fc) + C n +i(A; + l)/i(n, A; + 2) 

+ ^rJ2h(n,j)F(j,k,p). (59) 

Thus, (l34l ) holds for i = A; + 1 and the proof is completed by induction. 

E. Proof of Theorem IJ.il 

Proof: Define the Liapunov function V(x) = (x'x) /2 for x G R^ . Use E n to denote the conditional 
expectation with respect to the cr-algebra, % n , generated by {yj,8j, j < n}. 



V n {V(g n+1 ) - V(g n )} =E n [g' n [-eg n + e (y n+1 - E{g{6 n )}) + E{g(6 n ) - g(6 n+1 )}]} 

+ E n {| - eg n + e (y n+1 - E{g(0 n )}) + E{g(6 n ) - g{9 n+l )}\ 2 }, (60) 



where y n +i and g(9 n ) are vectors in 1* with elements y n {i) and g(9 n ,i), 1 < i < iVo, respectively. 
It is easily seen that 

E„{g(0„) - g(9 n+ i)} = 0(p), B n {\g(9 n ) - g(9 n+1 )\ 2 } = O(p). (61) 



Using K to denote a generic positive value (with the notation KK = K and K + K = K), a farmiliar 

2 



inequality ab < a2 + fc2 yields 



O(ep) = 0(e 2 + p 2 ). (62) 
Moreover we have \g n \ = \g n \ • 1 < (\g n \ 2 + l)/2. Thus 

0(p)\g n \<0( P )(V(g n ) + l). (63) 

Then detailed estimates lead to 

E n { I - eg n + e (y n +i - E{g(S„,)}) + E{s(0 n ) - s(0 re +i)} f} = 0(e 2 + p 2 )(^(?n) + 1) (64) 
Furthermore, wee obtain that 

E n {V(g n+1 ) - V(g n )} = - 2eV(g n ) + eB n {g' n [y n+1 - Bg(9 n )}} 

+ E n {g , n E[g(9 n+1 ) - g(9 n )}} + 0(e 2 + p 2 )(V(g n ) + 1). (65) 

Define Vf and V 2 as following 

00 

V p (g,n) = e^E n {y j+1 - Eg(0j)h 



j=n 
00 



V p (g, n) = J2 9'V n {g(9j) ~ 9(0j+i)h (66) 



It can be shown that 



Define W(g, n) as 



\Vf(g,n)\=0(e)(V(g) + l), 
\V p (g,n)\=0(p)(V(g) + l). 



(67) 



W(g, n) = V(g) + V p (g, n) + V p (g, n). (68) 

This leads to 

E n {W(g n+u n + 1) - W(g n ,n)} =E n {V(g n+1 ) - V(g n )} + E n {V p (g n+1 , n + 1) - V p (g n ,n)} 

+ E n {V p (g n+1 ,n + 1) - V p (g n , n)}. (69) 



Moreover, 

E n {W(g n+ i,n+ 1) - W(g n ,n)} = -2eV(g n ) + 0(e 2 + p 2 )(V(g n ) + 1). (70) 
Eq. (l70l can be rewritten as 

E n {W(g n+1 ,n + l) - W(g n ,n)} < -2eW(g n , n) + 0(e 2 + p 2 )(W(g n , n) + 1). (71) 
If e and p are chosen small enough, then there exists an small A such that — 2e + 0(p 2 ) + 0(e 2 ) < —Xe. 



So (1711 ) can be re-arranged to the following, 

B n {W(g n+1 ,n + l) < (l-\e)W(g n ,n) + 0(e 2 +p 2 ). (72) 
Taking expectation of both sides yields 

E{W(g n+1 ,n + l)} < (1- \e)E{W(g n ,n)} + 0(e 2 + p 2 ). (73) 
Iterating on d73l yields 

n 

E{W(g n+1 ,n + l)} < (l-\e) n - N >>E{W(g Np ,N p )}+ ^ 0(e 2 + p 2 ){\ - Xe)^, (74) 

j=N P 

so 

E{W(g n+1 ,n + l)} < (l-Xe) n - N >E{W(g Np ,N p )} + o(^e+?pj . (75) 
If n is large enough we can approximate (1 — Xe) n ~ Np = O(e) 

E{^(5f n+1 ,n + l)}<of £ + ^ (76) 



Finally, using (l67l ) and replacing W(g n +i,n + 1) with V(g n+ i), we obtain 



Sketch of the Proof of Theorem \3.2\ 

Since the proof is similar to [13, Theorem 4.5], we only indicate the main steps needed and omit most 
of the vabatim details. 

(1) First we show that the two component process (g 6 (■) , 9 e (■)) is tight in D([0, T] : 1* x M). Using 
the techniques as in [28, Theorem 4.3], it can be shown that 9 e {-) converges weakly to a continuous-time 
Markov chain generated by Q. Thus, we mainly need to consider g £ (-). We show that 

limlimsupE[ sup Ef \g e (t + s) - g e (t)\ 2 } = 0, 



where Ef denotes the conditioning on the past information up to t. Then the tightness follows from the 
criterion [29, p. 47]. 

(2) Since (g 5 (■) , 9 s (■)) is tight, we can extract weakly convergent subsequence according to the 
Prohorov theorem (see [12]). To figure out the limit, we show that (g 6 (■) , 9 £ (■)) is a solution of the 
martingale problem with operator Lq. For each i G M. and continuously differential function with 
compact support f(-,i), the operator is given by 

L f(g,i) = Vf'@,i)[-g + g(i)] + £ qij f(gJ), i e M. (78) 

jeM 

We can further demonstrate the martingale problem with operator Lq has a unique solution in the sense 
in distribution. Thus the desired convergence property follows. 

G. Sketch of the Proof of Theorem \3.3\ 

(1) First note 

n TP- fa \\ i E [^(^n) _ g(0n+l] / r 7Q\ 

u n+ i = v n - ev n + Ve(2/n+i - Eg(0 n )) H -= . (79) 

v e 

Again, the approach is similar to that of [13, Theorem 5.6]. So again, we will be brief. 

(2) Define an operator 

Cf(u,i) = -Vf'(v,i)v + ^tr[V 2 f(v,i)Z(i)] + £ <Hjf(u,j), i G M, (80) 

j£M 

for function /(•,£) that has continuous partial derivatives with respect to v up to the second order and 
that has compact support. It can be show that the associated martingale problem has a unique solution 
in the sense in distribution. 

(3) It is natural now to work with a truncated process. For a fixed but otherwise arbitrary r-y > 0, 
define a truncation function 

l, ifxeS ri , 
0, if x ER No -S ri , 
where 5 ri ={i£ M. No : \x\ < r±}. Then we get the truncated iterates 



ri (*) 



<Vi = < ~ e^Q^K 1 ) + V~e(y n+ i - Eg(6* n )) + /^V ^)- (81) 

Define ^ e,ri (t) = v r ^ for t G [en,e?i + e). Then z/ £,Tl (-) is an r-truncation of v £ {-); see [12, p. 284] for a 
definition. We then show the truncated process (i/ £,ri (•), 9 e {-)) is tight. Moreover, by Prohorov's theorem, 



we can extract a convergent subsequence with limit (y ri (•), #(•)) such that the limit (y Tl (•), #(•)) is the 
solution of the martingale problem with operator C' 1 defined by 

C r *r(v,i) = -Vr>'{v,i)v + itr[V 2 / ri (^i)S«] + J] Qur^J), i e M (82) 

where f Tl {v,i) = f(u,i)q Tl (u). 

(4) Letting ri — >• oo, we show that the un-truncated process also converges and the limit denoted by 
(z/(-),0(-)) is precisely the martingale problem with operator C defined in (l80l ). Furthermore, the limit 
covariance can be evaluated as in [13, Lemma 5.2]. 
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